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Abstract 


This  report  briefly  outlines  research  performed  under  a 
contract  awarded  for  investigating  processes  affecting  the 
accuracy  of  performance  appraisals.  A  general  overview  of  the 
research  is  provided,  followed  by  a  listing  of  major  topic  areas 
investigated  and  reference  to  the  research  reports  that  describe 
the  research  in  detail.  An  Appendix  to  the  report  lists  the 
titles  of  the  technical  reports,  presentations  at  professional 
meetings,  and  theses  or  dissertations  that  resulted  from  the 
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Personnel  Technology  -  Performance  Appraisal,  A  Process  Approach 

A  Final  Report 

Performance  appraisal  systems  play  a  central  role  in  the 
effective  functioning  of  any  large  organization.  However, 
despite  the  importance  and  the  increasing  demand  for  effective 
performance  appraisal  systems,  our  ability  to  perfect  these 
systems  has  been  quite  limited.  For  the  most  part,  work  on 
performance  appraisal  has  focused  on  (1)  the  design  of 
performance  appraisal  instruments  and  (2)  the  training  of  people 
to  use  the  scales.  In  order  to  advance  beyond  our  present  state 
with  regard  to  performance  appraisal,  we  must  resist  the 
tendency,  represented  by  these  two  approaches,  to  stress 
intervention  over  understanding.  Instead,  it  is  necessary  to 
have  a  better  understanding  of  the  processes  involved  as  one 
individual  judges  the  performance  of  another. 

The  purpose  of  this  research  project  was  to  adopt  a  process 
view  of  performance  appraisal  and  then  explore  and  test 
assumptions  from  this  framework  with  empirical  research. 
Originally,  we  intended  to  approach  this  goal  through  the 
development  of  a  three-phase  research  program,  which  was  to  be 
conducted  over  a  period  of  three  years.  During  Phase  I,  task 
materials  and  measures  necessary  for  assessing  rater  accuracy 
were  to  be  developed.  The  purpose  of  Phase  II  was  to  explore 
those  factors  that  influence  the  accuracy  of  performance 
appraisals.  The  third  phase  was  intended  to  focus  upon 
applications.  However,  it  became  necessary  to  terminate  this 
contract  after  only  one  year.  As  a  result,  research  described  in 
this  report  focuses  primarily  on  Phase  I  activities.  The 
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activities  planned  for  Phases  II  and  III  will  be  completed  under 
separate  contracts  to  Dr.  Barnes-Farrell  and  to  Dr.  Ilgen.  The 
research  which  will  be  conducted  under  those  contracts  represents 
the  continuation  of  work  begun  during  this  project. 

The  primary  focus  of  our  work  during  Phase  I  was  the 
development  of  task  materials  and  measures  necessary  for 
assessing  rater  accuracy.  Materials  were  developed  for  use  in 
laboratory  and  procedures  for  use  in  field  research  settings 
respectively. 

Laboratory  materials.  Our  goal  was  to  produce  a  set  of 
videotaped  samples  of  employee  performance  in  which  the  true 
level  of  performance  could  be  accurately  estimated  by  expert 
raters.  In  addition,  behavioral ly  oriented  performance  rating 
scales  had  to  be  developed  to  accompany  the  videotaped  stimulus 
materials.  These  materials  would  provide  use  with  a  means  of 
assessing  the  accuracy  of  performance  ratings  pro/ided  by 
evaluators  under  a  variety  of  controlled  laboratory  conditions. 

The  development  of  laboratory  materials  took  place  in  two 
major  stages,  each  consisting  of  several  tasks.  The  first  step 
was  to  identify  a  job  and  to  develop  an  instrument  for  evaluating 
performance  on  each  of  the  important  performance  dimensions 
included  in  that  job.  The  second  stage  was  the  production  of 
videotaped  samples  of  performance  which  provided  behavioral 
examples  of  performance  at  low,  moderate,  and  high  levels  of 
effectiveness  for  each  performance  dimension.  Briefly, they 
can  be  summarized  as  follows. 

A  job  type,  "restaurant  waiting"  was  identified  as  a 


position  for  the  development  of  a  laboratory  measure  of  rating 
accuracy.  That  job  was  selected,  in  part,  because  persons  used 
later  with  the  stimulus  materials  would  have  some  familiarity 
with  it,  because  relatively  objective  measures  could  be 
developed,  and  because  we  could  gain  access  to  a  facility  for 
creating  videtaped  episodes  of  performance.  Students  from  the 
Department  of  Restaurant,  Hotel,  and  Institutional  Management 
worked  with  our  research  team  to  identify  important  dimensions  of 
work  performance  and  examples  of  effective  and  ineffective  work 
performance.  After  being  subjected  to  a  retranslation  procedure, 
scale  values  were  estimated  for  these  examples,  and  behaviorally 
anchored  rating  scales  were  constructed  for  each  dimension. 

Seven  dimensions  were  selected  for  videotaping:  Menu 
Familiarity,  Knowledge  of  Beverage  Service  Procedures, 
Maintaining  Performance  Levels,  Memory,  Cooperation,  Work  Habits, 
and  Dinner  Bill  Activities.  For  each  of  those  dimensions, 
benchmarks  with  high,  moderate,  and  low  scale  values  were 
identified  and  brief  scripts  were  prepared  to  represent  them. 
Each  script  was  performed  and  videotaped  twice:  once  with  a  male 
actor  playing  the  role  of  the  focal  employee,  and  once  with  a 
female  playing  that  role.  Thus  a  total  of  forty-two  vignettes 
were  produced  as  a  result  of  this  effort.  The  final  step  in  this 
process,  the  verification  of  intended  performance  levels  and 
final  editing  of  the  tapes  will  be  completed  under  Contract 
IN00014-83-K-0757  (Dr.  Barnes-Farrell,  Principal  Investigator). 

Field  materials.  Our  goal  in  the  development  of  field 
measures  was  to  develop  and  evaluate  a  set  of  procedures  for 
assessing  the  relative  accuracy  of  a)  different  rating  sources. 


and  b)  different  individual  raters  in  a  field  setting.  In 
addition#  we  hoped  to  develop  a  more  general  measure  of  "rating 
skill"  which  would  differentiate  between  accurate  and  inaccurate 
appraisers.  The  major  activities  involved  in  this  project  are 
summarized  below. 

The  cooperation  of  univetsity  custodians  was  obtained,  and 
the  job  of  custodian  became  the  focus  of  the  development  of  field 
materials.  As  before,  the  first  step  was  to  identify  the  primary 
performance  dimensions  of  the  job.  The  researchers  worked 
closely  with  small  groups  of  custodians,  who  identified  the 
following  eight  dimensions  as  important  to  work  effectiveness: 
1)  Floors,  2)  General  Cleaning  and  Polishing,  3)  Restrooms,  4) 
Classrooms,  Offices  and  Laboratories,  5)  Personal  Safety,  6) 
Custodial  Maintenance,  7)  Security,  and  3)  People.  Behavioral 
examples  providing  benchmarks  for  each  dimension  were  generated, 
retranslated,  and  scaled  to  produce  a  behavioral ly  oriented 
performance  rating  instrument  for  the  job  of  custodian. 

Once  the  scales  were  developed,  the  custodians  participating 
in  the  study  were  rated  by  five  different  sets  of  raters: 
supervisors,  peers,  self,  area  workers,  and  expert  observers. 
The  relative  accuracy  of  these  sources  was  examined,  using  four 
indices  of  accuracy.  In  addition,  the  relative  difficulty  in 
accurately  appraising  performance  on  each  of  the  eight  dimensions 
was  examined. 

Finally,  several  measures  in  addition  to  performance  were 
obtained  in  order  to  explore  some  of  the  factors  that  may 
influence  responses  to  performance  appraisal  scales.  These 
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included  measures  of  rater-ratee  familiarity,  perceived  source 
accuracy,  perceived  patterns  of  past  and  future  performance,  and 
causal  attributions  about  performance. 

Final  work  on  the  field  measure  will  be  completed  under 
Contract  #N00014-K-83-0756  (Dr.  llgen.  Principal  Investigator). 

Other  Research.  In  addition  to  the  development  of  task 
materials  and  accuracy  measures,  several  empirical  studies 
examining  various  aspects  of  the  appraisal  process  were  conducted 
during  year  one  of  this  contract.  The  effects  of  such  Variables 
as  prototypicality  of  the  ratee's  behavior  and  purpose  of  the 
appraisal  upon  accuracy  of  evaluation  were  examined,  and  the 
process  of  occupational  age-typing  was  studied  (see  1-1,  1-3, 
III-l,  I1I-2,  IV— 1 ,  I V-2 ,  IV- 3  in  the  Appendix).  This  work  is 
still  underway,  but  preliminary  findings  suggest  that  the 
investigation  of  process  variables  will  yield  valuable 
information  about  the  way  the  task  of  performance  evaluation  is 
carried  out  in  organizations  and  the  kinds  of  conditions  which 
will  facilitate  accurate  performance  evaluation. 
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